Using Nix flakes + Cachix + GitLab CI

Help
1

I’m trying to use Cachix with GitLab CI and Nix Flakes. I thought I figured it out, but somehow when I run nix build it gives me warning: Git tree '/builds/software-garden/website' is dirty and then builds everything without using cache (~10 minutes). It happens even when I run git clean --force -dX right before nix build. Same commands on my laptop work as expected, i.e. cache is used and re-build is very fast.

Here is a job that demonstrates it: dist (#1128477267) · Jobs · Software Garden / Website · GitLab

Any ideas? Maybe @domenkozar can help? :wink:

2 Likes
2

Why don’t you just put a git diff into your pipeline just before the make result step to see what differs?

Also you want git clean -dfx. From man git clean:

       -x
           Don’t use the standard ignore rules (see gitignore(5)), but still use the ignore rules given with -e
           options from the command line. This allows removing all untracked files, including build products. This
           can be used (possibly in conjunction with git restore or git reset) to create a pristine working
           directory to test a clean build.

       -X
           Remove only files ignored by Git. This may be useful to rebuild everything from scratch, but keep
           manually created files.
3

Good point about git clean and diff. Unfortunately I didn’t learn anything new. The only thing different from the local repo is the owner of the files. It’s root in GitLab. Can this invalidate cache? But then if I run the same job twice it doesn’t use cache either. And it doesn’t seem like it’s pushing anuthing either.

$ git status
HEAD detached at 5cded4b
nothing to commit, working tree clean
$ git diff
$ ls -ARplh
.:
total 1M     
-rw-rw-rw-    1 root     root         399 Mar 25 17:47 .envrc
drwxrwxrwx    6 root     root        4.0K Mar 25 17:48 .git/
-rw-rw-rw-    1 root     root         134 Mar 25 17:47 .gitignore
-rw-rw-rw-    1 root     root         646 Mar 25 17:47 .gitlab-ci.yml
-rw-rw-rw-    1 root     root       34.3K Mar 25 17:47 LICENSE
-rw-rw-rw-    1 root     root        1.3K Mar 25 17:47 Makefile
drwxrwxrwx    2 root     root        4.0K Mar 25 17:47 assets/
-rw-rw-rw-    1 root     root        1.4K Mar 25 17:47 flake.lock
-rw-rw-rw-    1 root     root        1.1K Mar 25 17:47 flake.nix
-rwxrwxrwx    1 root     root         341 Mar 25 17:47 generate-redirects
-rw-rw-rw-    1 root     root         679 Mar 25 17:47 node-dependencies.nix
-rw-rw-rw-    1 root     root       19.9K Mar 25 17:47 node-env.nix
-rw-rw-rw-    1 root     root      365.8K Mar 25 17:47 node-packages.nix
-rw-rw-rw-    1 root     root          41 Mar 25 17:47 node-supplement.json
-rw-rw-rw-    1 root     root      488.5K Mar 25 17:47 package-lock.json
-rw-rw-rw-    1 root     root        1.0K Mar 25 17:47 package.json
drwxrwxrwx    2 root     root        4.0K Mar 25 17:47 src/
-rw-rw-rw-    1 root     root      172.0K Mar 25 17:47 supplement.nix
./.git:
total 64K    
-rw-rw-rw-    1 root     root         235 Mar 25 17:47 FETCH_HEAD
-rw-rw-rw-    1 root     root          41 Mar 25 17:47 HEAD
-rw-rw-rw-    1 root     root         281 Mar 25 17:47 config
-rw-r--r--    1 root     root        2.2K Mar 25 17:48 index
drwxrwxrwx    3 root     root        4.0K Mar 25 17:47 lfs/
drwxrwxrwx    3 root     root        4.0K Mar 25 17:47 logs/
drwxrwxrwx    4 root     root        4.0K Mar 25 17:47 objects/
drwxrwxrwx    6 root     root        4.0K Mar 25 17:47 refs/
./.git/lfs:
total 8K     
drwxrwxrwx    2 root     root        4.0K Mar 25 17:47 tmp/
./.git/lfs/tmp:
total 0      
./.git/logs:
total 16K    
-rw-rw-rw-    1 root     root         237 Mar 25 17:47 HEAD
drwxrwxrwx    3 root     root        4.0K Mar 25 17:47 refs/
./.git/logs/refs:
total 8K     
drwxrwxrwx    3 root     root        4.0K Mar 25 17:47 remotes/
./.git/logs/refs/remotes:
total 8K     
drwxrwxrwx    2 root     root        4.0K Mar 25 17:47 origin/
./.git/logs/refs/remotes/origin:
total 8K     
-rw-rw-rw-    1 root     root         331 Mar 25 17:47 cachix
./.git/objects:
total 16K    
drwxrwxrwx    2 root     root        4.0K Mar 25 17:47 info/
drwxrwxrwx    2 root     root        4.0K Mar 25 17:47 pack/
./.git/objects/info:
total 0      
./.git/objects/pack:
total 8M     
-r--r--r--    1 root     root        6.4K Mar 25 17:47 pack-036b0ae72ffc5e8cef96430d94470fc54b2243df.idx
-r--r--r--    1 root     root        8.4M Mar 25 17:47 pack-036b0ae72ffc5e8cef96430d94470fc54b2243df.pack
./.git/refs:
total 32K    
drwxrwxrwx    2 root     root        4.0K Mar 25 17:47 heads/
drwxrwxrwx    2 root     root        4.0K Mar 25 17:47 pipelines/
drwxrwxrwx    3 root     root        4.0K Mar 25 17:47 remotes/
drwxrwxrwx    2 root     root        4.0K Mar 25 17:47 tags/
./.git/refs/heads:
total 0      
./.git/refs/pipelines:
total 8K     
-rw-rw-rw-    1 root     root          41 Mar 25 17:47 276572757
./.git/refs/remotes:
total 8K     
drwxrwxrwx    2 root     root        4.0K Mar 25 17:47 origin/
./.git/refs/remotes/origin:
total 8K     
-rw-rw-rw-    1 root     root          41 Mar 25 17:47 cachix
./.git/refs/tags:
total 0      
./assets:
total 3M     
-rw-rw-rw-    1 root     root      103.7K Mar 25 17:47 helloquence-61189-unsplash.jpg
-rw-rw-rw-    1 root     root        2.4M Mar 25 17:47 riski-andriansyah-leafs-square-unsplash.jpg
-rw-rw-rw-    1 root     root      299.4K Mar 25 17:47 thomas-serer-1462714-unsplash.jpg
./src:
total 56K    
-rw-rw-rw-    1 root     root          75 Mar 25 17:47 analytics.pug
-rw-rw-rw-    1 root     root        3.0K Mar 25 17:47 business-developer.pug
-rw-rw-rw-    1 root     root         175 Mar 25 17:47 index.coffee
-rw-rw-rw-    1 root     root        2.8K Mar 25 17:47 index.pug
-rw-rw-rw-    1 root     root         815 Mar 25 17:47 redirect.html
-rw-rw-rw-    1 root     root         576 Mar 25 17:47 redirects
-rw-rw-rw-    1 root     root         266 Mar 25 17:47 style.scss
$ git clean -dfx
$ git status
HEAD detached at 5cded4b
nothing to commit, working tree clean
$ git diff
$ make result
cachix use software-garden
Configured https://software-garden.cachix.org binary cache in /etc/nix/nix.conf
nix --experimental-features "nix-command flakes" build
warning: Git tree '/builds/software-garden/website' is dirty
4

Can you put the git diff inside the Makefile after cachix use software-garden in the result rule?

1 Like
5

Nothing. dist (#1128723459) · Jobs · Software Garden / Website · GitLab

6

Maybe you are hitting this one

https://github.com/NixOS/nix/issues/4140

You could try a git reset --hard HEAD right before nix build and if that doesn’t work open a bug.

1 Like
7

I’m trying to replicate @edolstra’s issue but I’m getting

$ nix --experimental-features "nix-command flakes" flake info --json | jq '.url'
error: 'info' is not a recognised command
Try 'nix --help' for more information.
8

Nix in the nixos/nix:latest container is ancient (3 months is ancient by Nix standards). Use nixpkgs/nix-flakes:latest instead or nixpkgs/cachix-flakes:latest, then you also don’t have to install cachix.

Wait, you actually install nixFlakes so it’s recent enough. Maybe Eelco removed the nix flake info subcommand recently.

Yes, that’s what happened. It’s nix flake metadata now: Merge 'nix flake {info,list-inputs}' into 'nix flake metadata' · NixOS/nix@66fa1c7 · GitHub

1 Like
9

With nixpkgs/cachix-flakes I get this:

$ nix-env --install --attr nixpkgs.jq nixpkgs.git nixpkgs.gnumake
error: attribute 'nixpkgs' in selection path 'nixpkgs.jq' not found

Actually I’m getting it with any nixpkgs/... image.

Let me to revert to nixos/nix:latest and run the metadata command.

10

Yes, it looks similar to the issue you linked to, only git reset and git diff doesn’t seem to resolve the problem. From https://gitlab.com/software-garden/website/-/jobs/1128943915#L475 :

$ git reset --hard HEAD
HEAD is now at 4e01fc8 CI: Print all metadata, not just URL
$ git diff-index HEAD
$ nix --experimental-features "nix-command flakes" flake metadata --json | jq
warning: Git tree '/builds/software-garden/website' is dirty
{
  "description": "Software Garden website: https://software.garden/",
  "lastModified": 0,
  "locked": {
    "lastModified": 0,
    "narHash": "sha256-fORDWJoDaSRWxpTDdkjjtpIV9zRADX/1Nip+Xm9Igpo=",
    "type": "git",
    "url": "file:///builds/software-garden/website"
  },
  "locks": {
    "nodes": {
      "flake-compat": {
        "flake": false,
        "locked": {
          "lastModified": 1606424373,
          "narHash": "sha256-oq8d4//CJOrVj+EcOaSXvMebvuTkmBJuT5tzlfewUnQ=",
          "owner": "edolstra",
          "repo": "flake-compat",
          "rev": "99f1c2157fba4bfe6211a321fd0ee43199025dbf",
          "type": "github"
        },
        "original": {
          "owner": "edolstra",
          "repo": "flake-compat",
          "type": "github"
        }
      },
      "flake-utils": {
        "locked": {
          "lastModified": 1614513358,
          "narHash": "sha256-LakhOx3S1dRjnh0b5Dg3mbZyH0ToC9I8Y2wKSkBaTzU=",
          "owner": "numtide",
          "repo": "flake-utils",
          "rev": "5466c5bbece17adaab2d82fae80b46e807611bf3",
          "type": "github"
        },
        "original": {
          "owner": "numtide",
          "repo": "flake-utils",
          "type": "github"
        }
      },
      "nixpkgs": {
        "locked": {
          "lastModified": 1615449093,
          "narHash": "sha256-Twp4oj4XL104Oqnx+li3Frf4osEGwjCPtiipsgGq0tA=",
          "owner": "NixOS",
          "repo": "nixpkgs",
          "rev": "d211397507f5321d53fc6bf9282adb5aed5909ca",
          "type": "github"
        },
        "original": {
          "id": "nixpkgs",
          "type": "indirect"
        }
      },
      "root": {
        "inputs": {
          "flake-compat": "flake-compat",
          "flake-utils": "flake-utils",
          "nixpkgs": "nixpkgs"
        }
      }
    },
    "root": "root",
    "version": 7
  },
  "original": {
    "type": "git",
    "url": "file:///builds/software-garden/website"
  },
  "originalUrl": "git+file:///builds/software-garden/website",
  "path": "/nix/store/kmg5mzfiz3hcpbx8i9cjpmwxsy6g8k28-source",
  "resolved": {
    "type": "git",
    "url": "file:///builds/software-garden/website"
  },
  "resolvedUrl": "git+file:///builds/software-garden/website",
  "url": "git+file:///builds/software-garden/website"
}
11

Here is a partial solution thanks to @jojosch who helped me via Matrix chat:

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 383d80e..0caa9ce 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -13,18 +13,7 @@ dist:
         nixpkgs.cachix

   script:
-    - git diff-index HEAD
-    - git status
-    - git diff
-    - ls -ARplh
-    - git clean -dfx
-    - git diff-index HEAD
-    - git status
-    - git diff
-    - nix --experimental-features "nix-command flakes" flake metadata --json | jq
-    - git reset --hard HEAD
-    - git diff-index HEAD
-    - nix --experimental-features "nix-command flakes" flake metadata --json | jq
+    - nix --experimental-features "nix-command flakes" path-info --all | grep -v '\.drv$' | sort > /tmp/store-path-pre-build
     - make result
     - cp --recursive result/ dist/

diff --git a/Makefile b/Makefile
index d2dca35..d6b4c11 100644
--- a/Makefile
+++ b/Makefile
@@ -16,9 +16,8 @@ result: $(shell find src -type f)
 	$(nix) build --print-build-logs --verbose

 cache: result
-	$(nix) flake archive --json \
-	| jq --raw-output '.path, (.inputs | to_entries [] .value.path)' \
-	| cachix push software-garden
+	$(nix) path-info --all | grep -v '\.drv$$' | sort > /tmp/store-path-post-build
+	comm -13 /tmp/store-path-pre-build /tmp/store-path-post-build | cachix push software-garden

 .PHONY: develop
 develop: node-dependencies.nix node_modules

This propagates the cache from one machine (laptop) to another (GitLab CI) and it’s a big step forward and I really appreciate your help.

Unfortunately it doesn’t seem robust. For example in my first attempt I accidentally pushed a lot of unrelated paths including 400MB mesa build because I was running unrelated nix shell in another terminal. Wrong things can get pushed accidentally. This is not a problem if pushing from CI, because it’s containerized, but on development machine it is.

Also if I’m not mistaken, if I build the project and then build again and try to push the cache, nothing will get pushed, because on second attempt the paths are already in the store, right? And actually the most likely use case for me is building this project on my laptop before pushing to GitLab and saving time in CI.

So I want to figure out a way to identify and push the result of building this flake only. I thought that this is done as suggested by https://nixos.wiki/wiki/Flakes#Pushing_Flake_inputs_to_Cachix like that:

$ nix flake archive --json \
  | jq -r '.path,(.inputs|to_entries[].value.path)' \
  | cachix push $cache_name

So there is still some tinkering ahead of me :hammer_and_wrench:

Edit: does this make sense?

realpath result | cachix push software-garden
12

I think I found a solution I like. Essentially it’s this:

# Build using cache if available
cachix use ${cache-name}
nix build

# Push the results
nix path-info --recursive \
| cachix push ${cache-name}

# Push the inputs and sources
nix flake archive --json \
| jq --raw-output '.path, (.inputs | to_entries [] .value.path)' \
| cachix push ${cache-name}

I can run the above set of commands on my laptop (even before making a commit) and then in CI, and the cache will be used. This cuts the build time from ~10 minutes to ~2 and so far I’m not aware of any downsides, but I am still a novice at nix, so please let me know if there is something wrong with this approach.

Here is a makefile that implements this technique: Makefile · ba775291c37cd387862d9c163635539b32129bcb · Software Garden / Website · GitLab

It was @jojosch who pushed me on the right track by pointing out that the nix flake archive command only prints the source paths, not the result. Actually I’m not sure if pushing those paths to Cachix is useful at all, but it probably doesn’t hurt.

The warning: Git tree '/builds/...' is dirty was probably a red herring. I’m still getting this warning, but it doesn’t seem to prevent the use of cache.

Huge thanks to everyone who took effort to help me today, including @siraben and @domenkozar on Matrix chat and @hmenke here. I really appreciate the friendliness and wisdom of the Nix community.

5 Likes
13

I should spend some time writing good documentation for flakes+Cachix, I didn’t do it so far as I don’t recommend unreleased experimental features to newcomers, but here we are.

Great detective work @tad-lispy

5 Likes
14

I agree with you in not recommending the experimental features to newcommers, though there is a difference between recommending something and documenting edge cases related to it.

You can still put a big yellow box on top of that manual page that explains that flakes are an experimental feature and therefore no support beyond what is written on that page is given, while also explaining that due to the nature of experimental software the page might get outdated faster than one is able to update it.

1 Like
15

Agreed :slight_smile:

PS: some words to please discourse

2 Likes
16

I made dusk / gitlab-nix-flakes-ci · GitLab for this purpose. You can see it in action on .gitlab-ci.yml · master · dusk / test-nix-flakes-ci · GitLab

I’m not sure how it compares to the above solution, but I thought I’d share anyways.

3 Likes
17

Nice! Note that you can use the following command instead of the pre/post hooks:

cachix watch-exec mycache nix-build -- -j4

1 Like
18

Does it produce different result than

nix path-info --recursive \
| cachix push ${cache-name}
19

Those are two different modes:

  • watch-exec will push all paths that were built during the execution of the command that follows

  • nix path-info --recursive will list all paths in /nix/store